Text assisted telephony on wireless device method and apparatus

ABSTRACT

A communication system and method usable to facilitate communication between a hearing user and an assisted user. In particular, the system employs a wireless portable tablet or other portable electronic computing device linked to a captioning enabled phone as a remote interface for that phone, thereby providing an assisted user with more options, more freedom, and improved usability of the system.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/489,383, filed Sep. 29, 2021, which is a continuation of U.S. patentapplication Ser. No. 16/209,675, filed Dec. 4, 2018, and issued as U.S.Pat. No. 11,190,630 on Nov. 30, 2021, which is a continuation of U.S.patent application Ser. No. 15/285,641, filed on Oct. 5, 2016, whichclaims the benefit of priority to U.S. provisional application No.62/246,905, filed Oct. 27, 2015, each of which is incorporated herein byreference in its entirety.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not applicable.

BACKGROUND OF THE DISCLOSURE

The field of the disclosure is text assisted telephony and morespecifically telephony to help deaf or hard of hearing users receivecaptioned text from a relay on a personal portable device.

Systems have been developed to provide text captioning of voice calls toa deaf or hard of hearing person (hereinafter “an assisted user”) to aidthe assisted user in understanding words annunciated into a telephone bya remote hearing user. In at least some cases these systems rely on anassisted user having access to a captioned telephone that includes aspeaker, a microphone and a display screen and that is capable ofconnecting to first and second communication lines. Here, when a phonecall occurs using the captioned telephone linked to a first of thecommunication lines and an assisted user requires captioning, thecaptioned telephone is linked via the second line to a relay service anda hearing user's voice signal (e.g., the voice of the person linked tothe caption device via the first line) is provided to the relay service.A call assistant at the relay receives the voice signal of the hearinguser and converts that voice signal to a text stream which is sent backto the captioned telephone and presented to the assisted user via thedisplay screen. In at least some cases it is contemplated that automatedsoftware at the relay may convert some or all of the hearing user'svoice signal to text so that no call assistant is required.

In at least some cases the captioned telephone is equipped with arelatively large display screen (e.g., 7 to 10 inch diameter) so thatthe text presented on the screen can be relatively large to accommodatepersons that also have trouble seeing smaller text fonts. Captionedtelephones also often come with r speakers that can generate high volumesignals to broadcast a hearing user's voice signal to an assisted userwhen the assisted user has at least some ability to hear.

While captioned telephones have proven extremely useful and haveadvantageously changed the way deaf and hearing assisted userscommunicate, one problem with the captioned telephone solution is thatthe captioned telephone devices are somewhat expensive and thereforemost assisted users only have a single captioned telephone device. Incases where an assisted user only has a single captioned telephonedevice, the single device can only be located at a single location whichmeans that the assisted user has to travel to that device location inorder to use the captioning service. For instance, where an assisteduser's captioned telephone device is located in her kitchen, if theassisted user is located in her bedroom when a call is received or whenshe decides to place a call for which captioning is required, the userhas to travel to her kitchen in order to participate in the captionedcall. As another instance, if another assisted user is sitting on hiscouch watching a football game when a call is received at his captionedtelephone device that is located in his bedroom, the assisted user hasto get up and move to his bedroom to field the call.

While movement to the location of a single captioned telephone device issimply inconvenient for some assisted users, in other cases the task ofquickly travelling to a caption device when a call is received isexacerbated by other physical limitations. For instance, many assistedusers are elderly and may have difficulty quickly getting up from achair and moving to the captioned telephone location. Thus, here, when acall is received and an assisted user is sitting on a couch watching afootball game, the user may not be physically able to field the callquickly causing the caller to hang up. Even when a call is received andan assisted user is able to travel to the captioned telephone to answerthe call, as in the case of unassisted hearing users, an assisted usermay be extremely disappointed when an incoming call is answered (e.g.,in the case of a telemarketing call, a call from an acquaintance thatthe assisted user does not want to talk to when the call is received,etc.). Here, the act of getting up and travelling to the captionedtelephone to field what ends up being an unwanted call is disappointing.

In addition to the cost of a captioned telephone limiting the captionservice to one location in an assisted user's home, the cost typicallyresults in the assisted user not having access to a captioned telephoneat her office or place of work. Clearly, a hearing impaired person thatbenefits from a captioned telephone at home would also benefit from sucha phone at her place of work.

One solution that can be used to provide captioning service at almostall locations has been to provide captioning via a smart phone or otherportable device that includes a cellular or other type of communicationconnection. Here, a smart phone that can present text graphically on adisplay screen may be used in conjunction with other phone hardware toprovide near simultaneous voice and text to an assisted user.

While the portable smart phone or other device enables captioning atvirtually any location, this solution also has several shortcomings.First, this solution requires an assisted user to own a portable smartphone or similar device and to maintain a cellular voice and data planwhich can be prohibitively expensive for many people. Second, currentlyavailable portable smart phone devices have relatively poor audiocomponents so that a hearing user's broadcast voice signal is oftentimes too quiet or of poor quality which exacerbates the hearing problemfor many hearing assisted users. Third, the display screens on many cellphones and other portable devices are often times relatively small andtherefore not optimal for use by an assisted user that may require largefont. Fourth, where a cell phone or other portable device is usedinstead of a specialized captioned telephone device (e.g., a device withlarge screen, high quality loud speakers, etc.), the assisted user doesnot have the option to switch over use to a specialized captionedtelephone when that option is preferred.

Thus, there is a need for a captioning solution which enables hearingimpaired assisted users to obtain captioning of a hearing user's voicesignals at various locations within a home, office, or other workenvironment without requiring a cellular voice and data plan. In atleast some embodiments it would also be advantageous if the solutionwould allow an assisted user to use a captioned telephone device whenpreferred.

SUMMARY OF THE DISCLOSURE

It has been recognized that many captioned telephone device users orother persons that would benefit from captioned telephone servicesalready have access to devices that could be used to extend captionedtelephone service to locations that are remote from a typical captionedtelephone device. To this end, a smart phone, portable tablet typecomputing device, wrist mounted device, laptop or other portablecomputing device that includes a displays screen (hereinafter theassisted user's “portable device”) may be linked into a captionedtelephone communication network and used to present text and a hearinguser's voice to an assisted user as well as to obtain the assisteduser's voice and provide the assisted user's voice to the communicationnetwork to be sent to a linked hearing user. More specifically, anassisted user's captioned telephone device may be linked to a home oroffice intranet and the assisted user's portable device may bewirelessly linked to the home or office intranet or other WIFI system(hereinafter a “WIFI system”). Here, the assisted user's captionedtelephone device operates as a cordless caption phone (CCP) host deviceand the user's portable device operates as a CCP client device. When thehost device receives an incoming call, in addition to generating anin-coming call indication (e.g., generating an audible ring,illuminating an in-coming call light device, providing caller IDinformation on a host device display screen), the host device maywirelessly link via a home or office WIFI system to the client deviceand cause the client device to generate similar warning signals (e.g.,sound, light, caller ID information, etc.). In addition, the clientdevice may present a call answer icon or virtual control button via aclient device display screen which, when selected, causes the hostdevice to answer the in-coming call and commence a caption service.

Here, if an assisted user uses the host device to answer an in-comingcall on a first communication line in the usual fashion (e.g., bypicking up a host device receiver and without using a client device),the host device operates in a normal fashion. To this end, first, thehost device goes off hook to create a link on a first communication lineto the hearing user's device (e.g., the device used to initiate thecall). Second, the host device links to a relay via a secondcommunication line either automatically or when an assisted useraffirmatively requests caption service. The hearing user's voice signalreceived on the first line is transmitted to the relay on the secondline to be converted into text and the text is received back on thesecond line and presented on a host display screen. In addition, thehearing user's voice is broadcast to the assisted user via a speakerincluded in the host device. A host device microphone receives wordsspoken by the assisted user and transmits those words on the first lineto the hearing user's device to be broadcast to the hearing user.

In the alternative, if an assisted user answers an in-coming call usinga CCP client device (e.g., a tablet, smart phone or other portabledevice), the act of answering the call causes the client device totransmit a call answer signal to the host device via the WIFI systemwhich causes the host device to go off hook to create a link on a firstcommunication line to the hearing user's device (e.g., the device usedto initiate the call). Next, the host device links to a relay via asecond communication line either automatically or when the assisted useraffirmatively requests caption service via the client device (e.g., viaselection of a “Caption” icon presented on a client device displayscreen). Once captioning is to commence, the hearing user's voice signalreceived on the first line is transmitted to the relay on the secondline to be converted into text and the text is received back on thesecond line at the host device. The host device transmits the receivedtext along with the hearing user's voice signals to the assisted user'sclient device (e.g., the tablet) via the WIFI system. The client devicepresents the received text to the assisted user to be viewed and alsobroadcasts the hearing user's voice to the assisted user via a clientdevice speaker. A client device microphone receives words spoken by theassisted user and transmits those words via the WIFI system to the hostdevice which transmits those words on to the hearing user's device viathe first line to be broadcast to the hearing user.

In addition to being used to field in-coming calls, a CCP client and aCCP host system may be used to place outgoing calls. To this end, forinstance, an assisted user may use a client device to link to a hostdevice via a WIFI system and to control the host device to initiate acall to a hearing user's device. When a call is commenced, the hostdevice links to a hearing user's device via a first line and after thefirst link is made (or at least after the call is commenced), the hostdevice links to a relay via a second line to present hearing user voicesignals to the relay to be converted into text. The text and hearinguser's voice are transmitted via WIFI to the client device and presentedto the assisted user and the assisted user's voice signal is transmittedfrom the client device to the host device and then on to the hearinguser's device.

In other embodiments it is contemplated that a host device will presentcaptions via a host device display screen even if a client device issimultaneously being used to present captions. In this way, if anassisted user travels from a remote location to a host device during anongoing call, the assisted user can simply commence using the hostdevice display screen to read captions during the call. This may beadvantageous in cases where an assisted user prefers using a host deviceduring captioning because the device includes a relatively largerscreen, includes better speakers, is capable of a relatively louderbroadcast of the hearing user's voice signal, has adjustable soundcharacteristics (e.g., pitch and tones) that can be customized to auser's preferences and hearing capabilities, etc. In this case, someassisted user's may simply use a client device to field calls afterwhich the user would routinely move to the host device to participate inan ongoing call. Here, even after viewing captions on the host device,if an assisted user wants to move to a different location, the assisteduser can simply use the client device to follow along with the captionswhile traveling once at a remote location.

In some embodiments captions may only be provided via one or the otherof a host captioned device and an assisted user's client device at atime. For instance, where an assisted user answers an incoming callusing the client device, captions may be presented on that device alongwith a “Change Device” button or selectable virtual icon. Here, when theChange Device button is selected on the client device, the hostcaptioned device may start presenting text along with another “ChangeDevice” button or icon and the client device may stop presenting text.At this point, if the Change Device button is selected via the hostdevice, the client device may again start to present text along with aChange Device button and the host may cease presenting the text.

In most cases only one or the other of the host and a client device willpresent audio output of a hearing user's voice so that the audio is onlypresented once. In this way, an assisted user is not confused bypresenting audio two times where the audio is not precisely synced intime which would result in an annoying echo of the hearing user's voicesignal.

In some cases a client device may be linkable to other devices thatenhance a caption call such as a Bluetooth headset or a high qualitywireless speaker or set of speakers tuned to be optimized based on anassisted user's hearing capabilities. Hear, during a commissioningprocedure, the client device may cause a pre-recorded voice loop to beplayed via the wireless speakers and present an interface via the clientdevice screen that allows the user to adjust tone and pitch as well asperhaps other sound characteristics so that the assisted user can playwith the characteristic settings until optimized values are set.Thereafter, the user's settings may be used to optimize during anysubsequent caption calls. Other devices that may be linked to via aclient device to enhance a call include a neck loop, a hearing aid, orcochlear implant, etc.

In other embodiments, if an assisted user is using a client device toparticipate in a caption call and the assisted user picks up a hostdevice receiver, the host device may start presenting captions via thehost device display either instead of or in parallel with the clientdevice.

In at least some embodiments it is contemplated that a captioningapplication may be downloaded to any CCP client device which can be usedto perform the various functions required at the client device. Once theapplication is downloaded to the client device, a commissioning processassociated with the application may be performed to associate the clientdevice and specifically the captioning application with the host device.Thereafter, when a call is received at the host device, if the clientdevice is located within a range of the host device of an associatedwireless network, the host device links to the client device andcommunicates therewith as described above. Similarly, after the clientand host devices are associated, when the client device is used toinitiate a call via the captioning application and the host device, theclient device links via a WIFI system to the host device and the hostand client devices communicate as described above.

It has also been recognized that other communication devices within anassisted user's environment are suitable for presenting captioning to anassisted user during a caption call. For instance, many televisions nowinclude WIFI transceivers and allow a user to download variousapplications to a television based memory that can be run by atelevision based processor. Here, a captioning application can bedownloaded to a suitable television device and may be associated with ahost device during a commissioning procedure after which the televisiondevice operates in a fashion similar to that described above withrespect to the tablet based client device. Here, the television may needto be supplemented with some other devices such as, for instance, amicrophone and/or high quality speakers. In other cases a laptopcomputer or other computing device that has wireless communicationcapabilities, a display screen, a speaker and a microphone may operateas a client device.

In some cases, other pre-existing hardware may be useable with atelevision to facilitate communication. For instance, some companies arenow manufacturing hardware devices that link to televisions forreceiving wireless information for driving the linked television and insome of these cases a microphone is provided along with the devices. Forexample, the new Apple TV device includes a remote control that includesa microphone. Here, the microphone included with the remote may be usedby an assisted user to generate a voice signal intended for a hearinguser. The voice signal would be provided to the linked hardware devicewhich would then be wirelessly transmitted to the host device and on tothe hearing user's device.

In at least some cases two or more client devices may be associated witha single host device. For instance, an assisted user may download acaptioning application to each of a portable tablet device and a largescreen television device located in the user's living room and mayassociate each of the tablet and television devices with a host devicein the user's kitchen. Thereafter, when an in-coming call is received atthe host device, the host device may use the WIFI system to sendmessages to each of the tablet and television devices indicating theincoming call and an assisted user may be able to use either of thosedevices to answer the call and start a caption session.

In some cases where two or more client devices are associated with ahost device, one of the client devices may handle a first subset ofcommunication tasks while the other handles a second subset ofcommunication tasks. For instance, where both a large flat screentelevision and a tablet are associated with one host, text may bepresented via the large television display and an audio portion of acommunication may be handled via the tablet that includes speakers and amicrophone. For instance, here, when an incoming call is received, thetablet may indicate the incoming call and allow a user to accept thecall by selecting an on screen virtual icon. Once the call is answeredand captioning commences, the host may link to the television andpresent captions on the television screen and may maintain a link to thetablet to receive an assisted user's voice signal and to provide thehearing user's voice signal to the assisted user via broadcast using atablet speaker. As another instance wireless speakers may be associatedwith a host device and driven directly by the host device while a tabletor television display presents captions. Communication preferences maybe set up by an assisted user or a proxy (e.g., a family member)therefore during a device commissioning procedure.

In other cases it is contemplated that a host device will only associatewith one client device and the client device may then be linkable toother affordances (e.g., a large television display, wireless speakers,a wireless microphone, etc.) via Bluetooth or some other wirelesscommunication protocol.

In still other embodiments it is contemplated that a simplified hostdevice may be provided that does not include any of the human interfacefeatures of a typical captioned device like a display screen, a speaker,a microphone for receiving an assisted user's voice signals, etc. Here,the host device would simply operate as an intermediate device forcontrolling call information routing during in-coming and outgoing callsbetween a hearing user's device, a relay and a portable client device.In effect, the portable client device would operate as the only captionoutput and hearing user voice signal output device to the assisted userand as the only device for obtaining a voice signal from the assisteduser.

Many captioned telephone host devices that are already installed haveall of the hardware required to facilitate client device enhancedservices as described above. In these cases it is contemplated thatinstalled host devices may be “retrofitted” by simply updating softwareon those devices so that those devices operate with portable personalclient devices as described above. In this way existing hardwareincluding existing portable client devices may be leveraged to provideadditional functionality without substantial additional expense.

U.S. Pat. No. 8,917,821 which is titled “Device Independent TextCaptioned Telephone Service” and which issued on Dec. 23, 2014 isincorporated herein in its entirety by reference and describes a systemwhere a computer or other networked device can provide host typecaptioning services. In at least some embodiments it is contemplatedthat a desktop or laptop computer may be used with a tablet or otherwirelessly linked client device to provide enhanced portable devicecaptioning services like those described above in the context of acaptioned telephone device.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 is a depiction of an exemplary communication system that can beused to facilitate communication between a hearing user and an assisteduser;

FIG. 2 is a depiction of a captioned telephone device;

FIG. 3 depicts the components of a portable tablet or other type ofportable computing device;

FIG. 4 is a depiction of a portable tablet or other type of portablecomputing device receiving an incoming call;

FIG. 5 is a depiction of exemplary data transfer between a captionedtelephone device, a portable computing device, and an assisted user;

FIG. 6 is a depiction of a portable tablet or other type of portablecomputing device during ongoing captioning;

FIG. 7 is a depiction of a portable tablet or other type of portabledevice displaying an exemplary captioning application main screen shotthat can be used to access different captioning application features;

FIG. 8 is a depiction of the portable tablet or other type of portabledevice of FIG. 7 after a user selects the call option;

FIG. 9 is a depiction of the captioned telephone device of FIG. 2 in themiddle of a call with captioning turned off;

FIG. 10 is a depiction of a second exemplary communication system thatcan be used to facilitate communication between a hearing user and anassisted user;

FIG. 11 is a depiction of a third exemplary communication system thatcan be used to facilitate communication between a hearing user and anassisted user;

FIG. 12 is a depiction of another screenshot of a portable tablet orother type of portable computing device;

FIG. 13 depicts the components of a second portable tablet or other typeof portable computing device;

FIG. 14 is a depiction of a television in wireless communication with aportable tablet or other type of portable computing device, thetelevision configured to serve as a display for an assisted user; and

FIG. 15 is a depiction of a fourth exemplary communication system thatcan be used to facilitate communication between a hearing user and anassisted user.

DETAILED DESCRIPTION OF THE DISCLOSURE

The various aspects of the subject disclosure are now described withreference to the drawings, wherein like reference numerals correspond tosimilar elements throughout the several views. It should be understood,however, that the drawings and detailed description hereafter relatingthereto are not intended to limit the claimed subject matter to theparticular form disclosed. Rather, the intention is to cover allmodifications, equivalents, and alternatives falling within the spiritand scope of the claimed subject matter.

As used herein, the terms “component,” “system” and the like areintended to refer to a computer-related entity, either hardware, acombination of hardware and software, software, or software inexecution. For example, a component may be, but is not limited to being,a process running on a processor, a processor, an object, an executable,a thread of execution, a program, and/or a computer. By way ofillustration, both an application running on a computer and the computercan be a component. One or more components may reside within a processand/or thread of execution and a component may be localized on onecomputer and/or distributed between two or more computers or processors.

The word “exemplary” is used herein to mean serving as an example,instance, or illustration. Any aspect or design described herein as“exemplary” is not necessarily to be construed as preferred oradvantageous over other aspects or designs.

Furthermore, the disclosed subject matter may be implemented as asystem, method, apparatus, or article of manufacture using standardprogramming and/or engineering techniques to produce software, firmware,hardware, or any combination thereof to control a computer or processorbased device to implement aspects detailed herein. The term “article ofmanufacture” (or alternatively, “computer program product”) as usedherein is intended to encompass a computer program accessible from anycomputer-readable device, carrier, or media. For example, computerreadable media can include but are not limited to magnetic storagedevices (e.g., hard disk, floppy disk, magnetic strips . . . ), opticaldisks (e.g., compact disk (CD), digital versatile disk (DVD) . . . ),smart cards, and flash memory devices (e.g., card, stick). Additionallyit should be appreciated that a carrier wave can be employed to carrycomputer-readable electronic data such as those used in transmitting andreceiving electronic mail or in accessing a network such as the Internetor a local area network (LAN). Of course, those skilled in the art willrecognize many modifications may be made to this configuration withoutdeparting from the scope or spirit of the claimed subject matter.

Referring now to the drawings wherein like reference numerals correspondto similar elements throughout the several views and, more specifically,referring to FIG. 1 , the present disclosure will be described in thecontext of an exemplary communication system 10 that can be used tofacilitate communication between a hearing user 22 and an assisted user24 (e.g., a hearing impaired user) where the communication is similar tothe type of communication that occurs between two hearing persons via atelephone connection. The hearing user 22 may use a normal telephone tolink into the system via a PSTN or other type of telephone line orcommunication link to make a call or receive a call from the assisteduser 24.

Exemplary system 10 includes a captioned telephone device 14, a wirelessrouter device 16 and a tablet type computing device 20 located at theresidence or some other location (e.g., an office, place or work, etc.)associated with assisted user 24 and within wireless broadcast range ofrouter 16. In addition, system 10 includes a relay or call center 12that is typically remote from the location at which router 16 islocated.

In at least some embodiments, captioned telephone device 14 is a twoline type caption device that is linkable by phone (e.g., PSTN) or othercommunication link (e.g., internet) on a first line to the hearinguser's device 18. In at least some embodiments device 14 may be usedlike a normal telephone device to facilitate a conventional telephoneconversation between two hearing users, one using device 18 and theother using device 14. To this end, referring also to FIG. 2 , device 14includes, among other components, a keyboard 32 for entering a phonenumber or other information and a handset 34 that can be used to receivea hearing user's voice broadcast from a speaker and to receive a localuser's voice signals via a microphone built into the handset (notseparately labeled). In at least some embodiments, while notillustrated, device 14 includes a speaker and a microphone that can beused to broadcast a hearing user's voice signal and to receive anassisted user's voice signal to be transmitted on to the hearing user atdevice 18. In some cases the speaker can be set to have a relativelyhigh volume to help an assisted user hear the hearing user's voicesignal. Voice signal attributes other than volume can be adjusted toincrease a specific user's ability to hear (e.g., attributes like pitchand tone can be adjusted to meet specific and instantaneous userpreferences).

Referring still to FIG. 2 , in addition to the components describedabove, captioned telephone device 14 also includes a large (e.g., 7 to 9inch diameter) display screen 30 which can be used to interface withdevice 14 by providing screen shots of content and virtual tools (see35, 44, 42, etc.) as well as to provide text or captioning correspondingto words spoken by a remote hearing user 22 to an assisted user usingdevice 14. To this end, see the exemplary caption text at 38 thatcorresponds to words spoken by hearing user 22 in the present example.The text is presented substantially in real time (e.g., with less than1-4 seconds delay in most cases and often times faster) as a remotehearing user speaks. The exemplary display screen shot 35 in FIG. 2includes additional useful information including a header at 37 whichindicates the identity of the remote user 22 when available as well as acurrent time and date and a call timer which indicates the periodassociated with an ongoing call in minutes and seconds.

In at least some cases the screen 30 is touch sensitive so that virtualcontrol icons or buttons can be presented on the screen and the screencan be used to sense when a user selects one of the virtual buttons.Exemplary virtual buttons include a scrolling tool at 40 enabling a userto scroll through content as that content is presented on the screen 30,a pause button 44 selectable to pause the captioning text service and atext resizing tool at 42 useable to increase or decrease the size oftext presented in the captioned phrases at 38. In at least some casesthe keyboard may include a special key 36 that is selectable to startand stop the captioning service at any time during an ongoing call oreven prior to commencement of a call. For instance, while a conventionalvoice call is ongoing, an assisted user may recognize that she needscaptioning as she cannot make out at least some words uttered by ahearing user. In this case, the user may select button 36 to start acaptioning service. At some point while captioning is being providedduring an ongoing call, a user may recognize that she does not need thecaptioned text and may select button 36 to turn off the captioningservice.

Referring again to FIG. 1 , device 14 has an Ethernet port or a wirelesstransceiver that can be used to link to a local area network routerdevice 16 or some other intermediate device that in turn is linked tothe router device. The router 16 is linked via an internet or othercommunication link to the relay center 12. When captioning is notrequired, device 14 is, in at least some cases, not linked to router 16and instead conventional phone communications occur between a hearinguser's device 18 and the assisted user's device 14. When captioning isrequired, device 14 links to router 16 and causes a connection to relay12 over which the hearing user's voice signal is transmitted. Here, whendevice 14 is used without a remote tablet device 20, device 14 cancelsan assisted user's voice signal from the voice signals transmitted tothe relay 12 so that only the hearing user's voice signals arrive at therelay 12 to be converted to text. By cancelling the assisted user'svoice from the signal sent to the relay, the relay can transcribe theentire received voice signal without attempting to separate the assisteduser's voice from the hearing user's voice signal and the transcriptionprocess is rendered more simple.

Referring still to FIG. 1 , relay center 12 comprises a system that canreceive voice signals via the internet or other communication connectionand transcribe the voice signals into text. In at least some cases,center 12 may employ call assistants or relay operators for transcribingvoice signals to text. To this end, in at least some cases, each relaycall assistant will be equipped with a computer having a display screen,a keyboard, a head set including an earphone and a microphone andperhaps other input devices. The computer may operate voice-to-texttranscription software that is trained to the voice of the callassistant to transcribe voice signals from the call assistant to text.The hearing user's voice signal may be presented to the ear of the callassistant and the call assistant may revoice the hearing user's voicesignal into the headset microphone. The computer then uses thetranscription software to transcribe the call assistant's voice to textwhich is, in at least some cases, presented on the display screen linkedto the call assistant's computer. The call assistant can then correctany transcription errors in the text presented on the display. Thetranscribed text is transmitted back to router 16 via the internet orother communication link which transmits the text on to device 14 to bedisplayed on screen 30. In at least some cases the transcribed text istransmitted to device 14 only after error correction has occurred. Inother cases, the transcribed text is transmitted immediately upongeneration by the software including errors and then any corrections tothe transcribed text are subsequently transmitted and used to make inline or other corrections to the prior transmitted text so thattranscription is presented to the assisted user as quickly as possible.

In other cases, instead of call assistants transcribing the hearinguser's voice signal to text, automated transcription software running ona server or computer at relay 12 may receive a hearing user's voicesignal and automatically transcribe that signal to text. Here, theautomatically transcribed text may be sent immediately back to device 14via the internet link to be presented via device 14. In other cases, acall assistant may correct the software transcribed text and correctionsmay either be made prior to transmitting the software transcribed textto device 14 or after that text is transmitted in which case thecorrected text would be transmitted to device 14 and used to make inline or other types of corrections at device 14. In still other casesother combinations of automated transcription software and callassistant activities may be used at relay 12 to transcribe hearinguser's voice to text. U.S. patent application Ser. No. 14/632,257 whichis titled “Semi-automated Relay Method And Apparatus” and which wasfiled on Feb. 26, 2015 describes several different relay systems wheresoftware or a call assistant and software combination are used totranscribe a hearing user's voice to text, any of which may be used insystem 10, which is incorporated herein in its entirety by reference.

Referring yet again to FIG. 1 , consistent with that least some aspectsof the present disclosure, an assisted user 24 may own or have access toa portable tablet or other type of portable computing device 20. Here,device 20 may be any type of portable computing device that includes adisplay screen, a wireless transceiver and a processor that is capableof performing the methods and processes described herein. For instance,device 20 may include a laptop computer, a tablet type computing devicelike an I-Pad, an Android based tablet device, etc., a smart phone orother type of computing device. Hereinafter, unless indicated otherwise,while device 20 may take many different forms, the phrase “tabletdevice” will be used to refer to device 20 or any other type of portablecomputing device used in a system 10.

Referring to FIGS. 3 and 4 , exemplary tablet device 20 is illustratedwhich includes, among other components, a touch sensitive display screen50, a processor 63, a transceiver 65, one or more speakers 69, amicrophone 59 and, in at least some cases, one or more hardwareinterface buttons 67. Processor 63 is linked to each of the screen 50,transceiver 65, buttons 67, microphone 59 and speaker 69. Wirelesssignals from router 16 are received via transceiver 65 and wirelesssignals are also transmitted from processor 63 to router 16 viatransceiver 65. In addition, processor 63 can communicate with otherwireless devices such as an assisted user's headset 82 (see also FIG. 1) that includes earphones and a microphone to transmit a hearing user'svoice signal to and receive an assisted user's voice signal from device82, a wireless speaker or microphone, etc.

As seen in FIG. 1 , a wireless (e.g., WIFI) link may be establishedbetween router 16 and client device 20 so that device 20 can benetworked to the captioned telephone device 14 as well as to relay 12.Here, client device 20 can be used as a wireless remote interface tocaptioned device 14 and, in at least some cases, may supportsubstantially all or all of the assisted user interface input and outputfunctionality that is provided by host device 14. Thus, for instance,when a call is received at captioned device 14, in addition toannouncing the call at device 14 via a ring tone and some type of visualnotice on screen 30 or a flashing light (not shown) associated withdevice 14, device 14 may transmit a signal to router 16 and on to clientdevice 20 causing device 20 to announce the incoming call in a similarfashion. To this end, see, for instance, FIG. 4 where a screen shot 51includes an announcement window at 52 indicating that an incoming callhas been received by device 14. The announcement includes, wherepossible, the name of the remote hearing person calling device 14 and aphone number associated with the device 18 used to initiate the call(see 54 in FIG. 4 ). In addition, the announcement includes selectablevirtual on screen icons or buttons 56 and 58 for answering the call andignoring the call, respectively. Here, to answer an incoming call viadevice 20, a user simply selects icon 56 and to stop the announcementand end the call, the user simply selects icon 58.

In addition to announcing via window 52, device 20 may generate aringing or other signal via speaker 69. In addition, device 20 maycontrol the background of screen 50 that surrounds window 52 to clearlyindicate an incoming call by, for instance, generating a red backgroundcolor as indicated by the left up to right cross hatching shown in FIG.4 .

In other cases the incoming call notice may be more discrete such as asimple banner (see 53 in FIG. 4 ) at the top of the screen 50 toannounce an incoming call. In still other cases device 20 may generatedifferent types of incoming call announcements depending on how device20 is currently being used by an assisted user. For instance, if theassisted user is currently using an application on device 20 for someother purpose (e.g., reading a news article, playing a game, etc.), arelatively discrete announcement like banner announcement 53 may bepresented as the user, staring at the screen, should be able to noticethe announcement when it arrives. Here, if the user is not currentlyusing an application on device 20, the announcement may be less discretesuch as the window form shown at 52 in FIG. 4 to more strongly call theuser's attention to the incoming call. In at least some cases sensorsalready included in device 20 may be able to discern the type of noticeto present as a function of immediate use by a device user. Forinstance, an accelerometer or other sensor in device 20 may sense whenthe device is being moved and may discern instantaneous use by a usertherefrom. As another example, a camera in device 20 may obtain an imageof the user when an incoming call is received and the device 20processor 63 may analyze the image and determine that the user islooking directly at the screen 50 so that a more discrete banner orother type incoming call announcement can be generated.

When the device 20 user receives a call and selects icon 58 in FIG. 4 toignore the call, device 20 transmits a signal via router 16 to hostdevice 14 indicating that the user does not want to answer the call. Inresponse to the received ignore signal, host device 14 may link theincoming call to a voice mail or call answering service. Here, the callanswering service may answer the call, present a voice message to thecalling party 22 and request that the calling party leave a voice mailmessage. When a voice mail message is provided, the voice signal may insome cases be stored solely as a voice mail message. In other cases, thevoice mail message may be automatically presented to relay 12 andconverted to text. The text may be transmitted back to device 14 to bestored locally along with the voice mail for subsequent access by theassisted user 82. In other cases the voice and transcribed text may bestored via a server at relay 12 to be accessed subsequently by user 24.In some cases where voice mail transcription is supported, transcriptionmay occur immediately when a voice is recorded while in other cases thetranscription may occur at some subsequent time as immediatetranscription of a stored voice mail is not typically necessary (e.g.,transcription 10 minutes after a hearing user's voice signal is recordedis usually acceptable).

Because transcription of a voice mail is typically not time sensitive,in at least some cases, voice to text transcription software may be usedto generate an initial text version of a voice signal. Here, in somecases the initial text version may be presented to an assisted user whenthe user accesses the voice mail associated therewith. In some cases, ifthe initial version of the text is insufficient (e.g., clearly notaccurate or includes excessive errors), the assisted user may request amore accurate version at which point the original voice signal may bepresented to a call assistant at relay 12 for full call assistanttranscription or at least for correction of the initial text version.Corrected or call assistant transcribed text may then be transmittedback to the assisted user device 14 or device 20 to be presented to theassisted user.

When device 20 user receives a call and selects answer icon 56 in FIG. 4, a signal is transmitted back through router 16 to captioned hostdevice 14. The signal received at device 14 causes device 14 to initiatea link via router 16 to relay 12 to commence captioning in at least someembodiments. In addition, the signal received at device 14 causes device14 to answer the incoming call (e.g., go off hook) and starttransmitting voice signals between the hearing user's device 18 and theassisted user's tablet 20 via router 16. Here, the hearing user's voicesignal received at device 14 is routed through router 16 to each ofrelay 12 and tablet 20. When the hearing user's voice signal is receivedat relay 12, the signal is converted to text and the text is transmittedback to device 14 via router 16. In at least some cases text received atdevice 14 will always be presented via device 14. In other cases wherean assisted user is using device 20 to view text, the text will not bepresented via device 14.

When device 14 receives text from relay 12 and an assisted user is usingtablet 20 for captioning, device 14 transmits the received text viarouter 16 to device 20. Thus, device 20 receives both the hearing user'svoice signal as well as the transcribed text associated with the hearinguser's voice signal. Because it takes at least some time to transcribethe voice signal to text, the hearing user's voice signal is typicallyreceived at tablet 20 at least some time prior to receiving the text andtherefore there is some delay between voice broadcast and textpresentation by tablet 20. For this reason, in at least some casesdevice 14 or tablet 20 or even relay 12 in some cases may delay thevoice signal for a short duration (e.g., 2 or less seconds) so that thetext and associated voice signal are more synchronized. In still othercases it is contemplated that time stamps may be associated withincoming voice signals (e.g., a separate time stamp for each worddiscerned in a received voice signal) and text may be associated witheach time stamp so that substantially precisely synchronized text andvoice can be presented. Here, delays in voice and text presentation maybe presented at the ends of sentences as opposed to between words sothat the sentences make more sense and can be presented at the pace ofthe original utterances by the hearing user.

Voice signals from assisted user 24 received via microphone 59 aretransmitted from tablet 20 via router 16 to device 14 and on to thehearing user's device 18 to be broadcast to the hearing user.

Thus, referring to FIG. 5 , when client device 20 is used as a remotecaptioning device, an additional IP connection 80 via a home or businessnetwork (e.g., a router 16) is established to initiate a two way audiostream (e.g., PCM via UDP), one way captioning to device 20 via TCP (orsome other suitable protocol) and two way command/control signaltransmission between host device 14 and client device 20.

Referring yet again to FIG. 1 , in at least some embodiments theassisted user 24 may use a wireless or tethered headset 82 includingearphones and a microphone. In this case, the assisted user's voicesignal would be obtained via the headset microphone and transmitted totablet 20 to be forwarded on to device 14 and then on to the hearinguser's device 18. The hearing user's voice signal would be transmittedfrom tablet 20 to the headset earphones to be broadcast to the assisteduser. Text from relay 12 would still be presented via the tablet 20display.

Referring now to FIG. 6 , an exemplary screen shot that may be presentedvia tablet screen 50 during ongoing captioning is shown. The exemplaryshot includes information and tools that are similar to the informationpresented via the captioned device screen shown in FIG. 2 . To this end,information descriptive of an on going call is presented at 37 aincluding the name of the hearing user participating in the call, thehearing user's telephone number and, where available, an image of thehearing user. Transcribed text is presented at 38 a and a text scrollingtool is presented at 30 a. Pause and text size control icons arepresented at 44 a and 42 a, respectively. Comparing the screen contentin FIG. 6 with the content in FIG. 2 , similar content is indicated bythe same numerical label, albeit including an “a” qualifier in FIG. 6 .By presenting content in a similar format and with a similar look andfeel, user's familiar with the content presentation on the captioneddevice screen 30 will be automatically familiar with the similarlypresented content on tablet screen 50.

Referring still to FIG. 6 , in addition to the other informationpresented as described above, screen shot 77 also includes an “End Call”icon 40 for ending an ongoing call. When icon 40 is selected, tablet 20may exit a captioning application immediately. In addition, upon endinga call, tablet 20 transmits a control signal to caption device 14causing device 14 to terminate the telephone link to the hearing user'sdevice 18 and terminate the link to relay 12.

In addition to using tablet 20 to field an incoming call, tablet 20 maybe used to place a call that provides captioning service. To this end,see FIG. 7 where an exemplary captioning application main screen shot 61is shown that includes several different virtual control icons 62, 64,66, 68 and 70 that can be used to access different captioningapplication features. Icon 62 is a “Call” icon which, when selected,allows an assisted user to initiate a captioned call to another person(e.g., the hearing user 22 in FIG. 1 ). For instance, when icon 62 isselected, in at least some embodiments the exemplary screen shot 71shown in FIG. 8 may be presented. Exemplary screen shot 71 includes afield 72 and a virtual numeric keyboard 74 for entering a telephonenumber. In addition, in at least some cases contact information forother persons that the assisted user has previously stored or hasrecently communicated with may be presented as at 78. In FIG. 8 theexemplary contact information includes a name and image of teach ofseveral different contacts. Any of the images may be selected to placean associated phone number in field 72 as an initial step to commencingan outgoing call. Other formats for presenting contact information 78are contemplated. Once a number is entered into field 72, phone icon 76may be selected to initiate a call.

Referring again to FIG. 1 , to initiate a call, device 20 transmits acall request via router 16 to caption device 14. Upon receiving the callrequest, device 14 dials out on the telephone line to the target hearinguser's device (e.g. the phone associated with the number entered orselected by the assisted user). In at least some cases if a hearing useranswers an outgoing call, caption device 14 automatically controlsrouter 16 to link to relay 12 to provide the hearing user's voice signalto relay 12 and to receive transcribed text back from relay 12 to bepresented to the assisted user. The returned transcribed text receivedat device 14 is routed to tablet 20 via router 16 to be presented to theassisted user 24. Again, the call can be terminated by the assisted userat any time by selecting the end call icon 40 shown in FIG. 6 .

Referring again to FIG. 7 , answering machine icon 64 may be selected toaccess stored voice mails and associated transcribed text. In at leastsome embodiments voice mails and associated text may be stored in adatabase linked to host device 14 or accessible via router 16. Historyicon 68 may be selected to access a list of recent calls or callattempts to render those contacts easily accessible. Favorites icon 70can be selected to access a list of an assisted user's favorite contactsfor convenience. Contacts icon 66 may be selected to access storedcontact information. In some cases the contact information is stored bya relay or other system server or device that is separate from portabletablet 20 so that selection of icon 66 causes

In at least some cases it is contemplated that an assisted user will notneed or want captioning when a call is initially answered but may wantto turn on a captioning service at some time during an ongoing call.Nevertheless, in this case, the assisted user would often like theability to answer an incoming call or initiate an outgoing call at alocation remote from the host device 14. For this reason, in at leastsome cases, calls may be answered or initiated using client device 20linked to host device 14 without captioning and the captioning may onlybe turned on at some subsequent time. To this end, if the answer icon 56is selected in FIG. 4 , at least initially, client device 20 may send asignal via router 16 to host device 14 to answer the call and start anaudio only communication session while presenting a “Caption” iconoption 99 as shown in FIG. 9 . Here, as the audio only call isprogressing, an assisted user may select icon 99 to cause host device 14to link via router 16 to relay 12 to commence the captioning service.Again, text would be transmitted via router 16 to host device 14 andfrom host device 14 through the router 16 to the client device to bepresented on the client device screen 50.

In other cases, when a tablet 20 is paired with or associated with acaption device 14, device 14 may tag a hearing user's voice signal witha virtual ID associated with tablet 20 so that when text is transmittedback from relay 12 to be presented to the assisted user, the text can besent through the router 16 directly to tablet 20 via virtual IDassociated therewith without having to first be received at the hostdevice 14. In this case, the text may or may not be transmitted todevice 14 and may or may not be presented via device 14 even iftransmitted thereto. Thus, for instance, when a call is received atdevice 14, an incoming call announcement (see 52 in FIG. 4 ) may bepresented via device 20 and when answered, device 14 may link to relay12 to start the captioning process. Here, the hearing user's voice istransmitted to relay 12 along with the assisted user's tablet virtual IDor address and relay transmits the transcribed text via the internet orother communication link through router 16 directly to tablet 20 to bepresented via screen 50.

In still other cases, although not shown, the host device 14 may includea built in router so that the Ethernet/wireless link shown in FIG. 1 isnot required. Here, host device 14 would simply link via the internet orother communication link to relay 12 and wirelessly to device 20 via alocal area or wide area network.

In yet other cases device 20 may include a personal hotspot or othertype of a router capability and router 16 may again not be needed. Tothis end, see FIG. 10 that is similar to FIG. 1 , albeit where therouter 16 is replaced by a router internal to device 20. In this case,when relay services are required and an assisted user is using a tabletor other portable device 20 with a built in routing capability, device14 may provide the hearing user's voice signal to device 20 which inturn transmits that voice signal to relay 12 while also broadcastingthat voice to the assisted user 24. Relay 12 again provides transcribedtext which is transmitted to tablet 20 to be displayed on the tabletscreen 50. The assisted user's voice signal in this case is alsotransmitted from tablet 20 to caption device 14 and then on to thehearing user's device 18. Here, if an assisted user wants to switch overto viewing captions on host device 20, the user may initiate thatprocess which would in turn cause captions to also be presented via thehost device display. Here, the phone link between the assisted user'shost device and the hearing user's device is a direct phone, internet orother type of communication link while device 20 handles back endcommunication with relay center 12.

In still other cases, instead of providing a full featured hostcaptioned device, a relatively less expensive captioned device may beprovided so that communication components that are provided via theclient device 20 are not duplicated via the host device. For instance,see FIG. 11 that is similar to FIG. 1 except that the full featuredcaptioned device 14 has been replaced by a captioned device brickcomponent 100 (hereinafter “brick device 100”). Brick device 100 has allof the components and capabilities of host device 14 except for thedisplay screen and handset (including eh speakers and microphone). Here,brick device 100 has an assigned phone number that can be used toreceive an incoming call from a remote user's device 18. As in the caseof host device 14 described above, device 100 can be controlled to linkvia a home network including router 16 to client device 20 to facilitatean audio only conversation between users of devices 18 and 20 or anaudio/captioned communication session between device 18 and 20 users.For instance, when an initial call comes in to brick device 100, device100 may link to client device 20 via router 16 and cause device 20 toannounce the incoming call (see again FIG. 4 ). When the call isanswered (e.g., icon 56 is selected in FIG. 4 ), brick device 100 mayfacilitate an audio only communication between the device 18 and 24users at least initially, client device 20 presenting a Caption buttonoption as in FIG. 9 . If button 99 is selected, client device 20 wouldsend a command to host brick device 100 to transmit the hearing user'svoice signal from device 18 to relay 12 via router 16 and the secondlink (e.g., the Internet link In FIG. 11 ). Relay 12 transmits captionedtext back to the brick device 100 which in turn transmits that text toclient device 20 via router 16 to be presented on the client devicedisplay screen 50. In this case, because brick device 50 does notinclude a display, speakers or a microphone, the assisted user does nothave the option to use device 100 as a direct input and output interfaceand instead would only use the client device 20 for those purposes.

In at least some cases a host device 14 and client device 20 will beused in a home or other facility where more than one phone device sharesthe phone line used by the host device 14 to link to remote user devices(e.g., 18). In this case, in at least some cases host device 14 willmonitor when any phone linked to the incoming line if off hook and maytransmit a signal indicating the off hook state to client device to beannounced (e.g., presented as a visual queue) to a user of device 20 sothat the device user understands the current state of the in comingline. Here, for instance, if the wife of an assisted user is using adifferent phone in the assisted user's home to participate in a call,the assisted user may not want to disturb that call and therefore thenotice of phone line state would be desired.

In at least some cases it is contemplated that an assisted user may joinan ongoing call that involves some other phone device in the assisteduser's home or other associated space using a client device 20 and theassisted user may or may not add captioning to the joined call. Forinstance, in some cases when an ongoing call is progressing, clientdevice 20 may present the option to “Join Call” via the device displaywhich, when selected, causes client device 20 to transmit a command tohost device 14 to go off hook and connect the audio portion of theongoing call to device 20 via router 16. If captions are not startedimmediately, when the assisted user wants to start the caption service,the user may select a caption icon as in FIG. 9 as described above.

Support case where text presented on tablet 20 and then moved to device14.

Support case where text presented via phone device remotely and thenwhen proximate device, 14, presenting text automatically via device 14to take advantage or larger screen, speaker, etc.

While the invention may be susceptible to various modifications andalternative forms, specific embodiments have been shown by way ofexample in the drawings and have been described in detail herein.However, it should be understood that the invention is not intended tobe limited to the particular forms disclosed. For example,

Thus, the invention is to cover all modifications, equivalents, andalternatives falling within the spirit and scope of the invention asdefined by the following appended claims.

To apprise the public of the scope of this invention, the followingclaims are made:

What is claimed is:
 1. A computer-implemented method to providecaptions, the method comprising: receiving audio data at a first devicefrom a second device over a first network, the audio data received bythe second device over a second network from a remote device thatgenerates the audio data, the second network is a different type ofnetwork than the first network; presenting, by the first device, audiobased on the audio data; and directing the audio data to a transcriptionsystem over a third network, wherein the transcription system isconfigured to generate captions of the audio data.
 2. The method ofclaim 1 further including obtaining, at the first device, the captionsfrom the transcription system, and presenting, by the first device, thecaptions.
 3. The method of claim 2 wherein the first device obtains thecaptions over the third network.
 4. The method of claim 3 wherein thethird network is the Internet.
 5. The method of claim 4 wherein thefirst network is a wireless network.
 6. The method of claim 5 whereinthe second network is a cellular network.
 7. The method of claim 2wherein the first device includes a computer including a display screenand wherein the captions are presented via the display screen.
 8. Themethod of claim 7 wherein the captions are presented substantiallycommensurate with presenting the audio.
 9. The method of claim 1 whereinthe first device transmits the audio data to the transcription systemover the third network.
 10. The method of claim 1 wherein the seconddevice transmits the audio data to the transcription system over thethird network.
 11. The method of claim 1 wherein the third network is adifferent type of network than the first and second networks.
 12. Acomputer-implemented method to provide captions, the method comprising:at a first device including a display screen and an audio output device,(i) receiving audio data from a second device over a first wirelessnetwork, the audio data received by the second device over a secondnetwork from a remote device that generates the audio data, the secondnetwork is a different type of network than the first network; (ii)broadcasting audio via the audio output device based on the audio data;(iii) directing the audio data to a transcription system over a thirdnetwork that is of a different type than the first network and thesecond network, wherein the transcription system is configured togenerate captions of the audio data; (iv) receiving the captions fromthe transcription system over the third network; and (v) presenting thecaptions via the display screen.